{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 用户社交数据(user_friends.csv)处理\n",
    "（只取训练集和测试集中出现的用户 ID）\n",
    "\n",
    "数据来源于 Kaggle 竞赛: Event Recommendation Engine Challenge, 根据\n",
    "①events they’ve responded to in the past\n",
    "②user demographic information\n",
    "③what events they’ve seen and clicked on in our app\n",
    "预测用户对某个活动是否感兴趣\n",
    "\n",
    "竞赛官网:\n",
    "https://www.kaggle.com/c/event-recommendation-engine-challenge/data\n",
    "\n",
    "user_friends.csv 文件: 共 2 维特征  \n",
    "user: 用户 ID\n",
    "friends: 以空格隔开的用户好友 ID 列表"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入工具包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import scipy.sparse as  ss\n",
    "import scipy.io as sio\n",
    "\n",
    "import pickle\n",
    "\n",
    "from sklearn.preprocessing import normalize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总的用户数目超过训练集和测试集中的用户, \n",
    "为节省处理时间和内存, 先去处理 train 和 test, 得到竞赛需要用到的活动和用户. \n",
    "然后对在训练集和测试集中出现过的活动和用户建立新的 ID 索引. \n",
    "先运行 user_event.ipynb, \n",
    "得到活动列表文件: PE_userIndex.pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取之前统计好的测试集和训练集中出现过的用户"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of users in train & test: 3391\n"
     ]
    }
   ],
   "source": [
    "# 读取训练集和测试集中出现过的用户列表\n",
    "userIndex = pickle.load(open('PE_userIndex.pkl', 'rb'))\n",
    "n_users = len(userIndex)\n",
    "print('Number of users in train & test: %d' %n_users)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取之前用户-活动分数矩阵, 将朋友参加活动的影响扩展到用户"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 用户-活动关系矩阵\n",
    "userEventScores = sio.mmread('PE_userEventScores')\n",
    "# 后续用于将用户朋友参加的活动影响到用户\n",
    "eventsForUser = pickle.load(open('PE_eventsForUser.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## user_friends.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 读取数据\n",
    "\"\"\"\n",
    "  找出某用户的那些朋友\n",
    "  1)如果你有更多的朋友, 可能你性格外向, 更容易参加各种活动\n",
    "  2)如果你朋友会参加某个活动, 可能你也会跟随去参加一下\n",
    "\"\"\"\n",
    "# 用户有多少个朋友\n",
    "numFriends = np.zeros((n_users))\n",
    "userFriends = ss.dok_matrix((n_users, n_users))\n",
    "\n",
    "fin = open('user_friends.csv', 'rb')\n",
    "fin.readline() # 跳过表头\n",
    "\n",
    "for line in fin: # 对每个用户\n",
    "    cols = line.strip().split(','.encode(encoding='utf-8'))\n",
    "    user = cols[0]\n",
    "    \n",
    "    if userIndex.__contains__(user):\n",
    "        friends = cols[1].split() # 朋友以空格隔开\n",
    "        i = userIndex[user] # 该用户的索引\n",
    "        numFriends[i] = len(friends)\n",
    "        for friend in friends: # 对该用户的每个朋友\n",
    "            str_friend = str(friend).encode(encoding='utf-8')\n",
    "            if userIndex.__contains__(str_friend): # 如果朋友也在训练集或测试集中出现\n",
    "                j = userIndex[str_friend]\n",
    "                \n",
    "                eventsForUser = userEventScores.getrow(j).todense()\n",
    "                \n",
    "                # 所有朋友参加活动的数量(平均频率)\n",
    "                score = eventsForUser.sum() / np.shape(eventsForUser)[1]\n",
    "                userFriends[i, j] += score\n",
    "                userFriends[j, i] += score\n",
    "                \n",
    "fin.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 用户的朋友数目\n",
    "# 归一化数组\n",
    "sumNumFriends = numFriends.sum(axis=0)\n",
    "numFriends = numFriends/sumNumFriends\n",
    "sio.mmwrite('UF_numFriends', np.matrix(numFriends))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "userFriends = normalize(userFriends, norm='l2', axis=0, copy=False)\n",
    "sio.mmwrite('UF_userFriends', userFriends)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  1.26494857e-04,   7.63793098e-05,   2.69069569e-04, ...,\n",
       "         8.64024193e-04,   2.92653356e-04,   3.12485176e-04])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numFriends"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
